healthchecker: use kube-proxy /livez to avoid scale-down false alerts#1306
healthchecker: use kube-proxy /livez to avoid scale-down false alerts#1306dpacgdm wants to merge 2 commits into
Conversation
Cluster Autoscaler marks nodes with ToBeDeletedByClusterAutoscaler during scale-down. kube-proxy /healthz intentionally fails in that case for LB connection draining (KEP-3836), which made NPD report kube-proxy as unhealthy. Switch the kube-proxy probe to /livez, which reflects process health only. Signed-off-by: dpacgdm <dpac.gdm@gmail.com>
kubernetes-prow
Bot
added
the
cncf-cla: yes
kubernetes-prow
Bot
added
the
needs-ok-to-test
|
Hi @dpacgdm. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
kubernetes-prow
Bot
added
the
size/S
|
Friendly ping: this PR is still waiting on |
hakman
left a comment
hakman
left a comment
There was a problem hiding this comment.
Looks good overall. Small comment nit: /livez still checks dataplane synchronization health, it only excludes node eligibility. Could we adjust the wording accordingly?
| kubeProxyHealthCheckEndpoint = fmt.Sprintf("http://%s/healthz", net.JoinHostPort(hostAddress, kubeProxyPort)) | ||
| // Use /livez for kube-proxy so Cluster Autoscaler scale-down (ToBeDeletedByClusterAutoscaler | ||
| // taint) does not make NPD report kube-proxy as unhealthy. /healthz intentionally fails in | ||
| // that case for load-balancer connection draining (KEP-3836); /livez only reflects process health. |
There was a problem hiding this comment.
| // that case for load-balancer connection draining (KEP-3836); /livez only reflects process health. | |
| // that case for load-balancer connection draining (KEP-3836); /livez reports kube-proxy | |
| // dataplane health without considering node eligibility. |
|
/ok-to-test |
kubernetes-prow
Bot
added
ok-to-test
|
Thanks — will fix the wording. You're right: |
/livez still checks process and dataplane sync health; it only excludes the node-eligibility check used by /healthz for scale-down draining. Signed-off-by: dpacgdm <dpac.gdm@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: dpacgdm The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
2 participants
Summary
Cluster Autoscaler marks nodes with
ToBeDeletedByClusterAutoscalerduring scale-down. kube-proxy/healthzintentionally fails in that case for load-balancer connection draining (KEP-3836), which caused NPD to report kube-proxy as unhealthy and fire unnecessary alerts.Switch the kube-proxy health check endpoint from
/healthzto/livez./livezstill checks process and dataplane synchronization health; it only excludes the node-eligibility check used for scale-down draining.Related issue
#1176
Test plan
go test ./pkg/healthchecker/... ./cmd/healthchecker/.../livezfor kube-proxy